ZERO-SHOT CROSS-LINGUAL TRANSFER

Cross-lingual Similarity of Multilingual Representations Revisited (2022.aacl-main)

Challenge:	Recent pruning methods remove redundant parameters according to parameter sensitivity, a gradient-based measure reflecting the contribution of the parameters.
Approach:	They propose a general task-agnostic method to balance parameter sensitivity and a novel adaptive learning method to control strength of intra-distillation loss for faster convergence.
Outcome:	The proposed method can reduce redundant parameters by over 80% without obvious performance degradation.

Copied to clipboard

Maksym Del, Mark Fishel

Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages (2022.acl-long)

Challenge:	Similarity indexes like CKA and CCA are not suitable for cross-lingual learning analysis.
Approach:	They propose an alternative that is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context.
Outcome:	The proposed method is exempt from the difficulties of CKA/CCA and is good specifically in a cross-lingual context.

Copied to clipboard

Vaidehi Patil, Partha Talukdar, Sunita Sarawagi

Low-Resource NLP

Challenge:	Pre-trained multilingual models have shown great potential for zero-shot cross-lingual transfer to low web-resource languages (LRLs).
Approach:	They propose a vocabulary generation algorithm which enhances lexical overlap across related languages by generating a token that increases the representation of LRLs.
Outcome:	The proposed approach improves cross-lingual transfer accuracy without reducing HRL representation and accuracy.

mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024.findings-eacl)

Copied to clipboard

Peiqin Lin, Chengzhi Hu, Zheyu Zhang, Andre Martins, Hinrich Schuetze

Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers (2021.eacl-srw)

Challenge:	Recent multilingual pretrained language models encode strong language-specific signals, which are not explicitly provided during pretraining.
Approach:	They propose a language similarity measure that induces similarities across languages from mPLMs using multi-parallel corpora.
Outcome:	The proposed measure exhibits moderately high correlations with linguistic similarity measures, and more accurate similarity results on low correlation languages.

Copied to clipboard

Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, Veronika Laippala

Cross-Lingual Transfer

Challenge:	Existing studies on register classification for web documents have limited results due to skewed datasets and low performance.
Approach:	They propose two new register-annotated corpora for French and Swedish . they show that deep pre-trained language models perform strongly in these languages .
Outcome:	The proposed models outperform existing models in English and Finnish and can match or surpass existing models.

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (D19-1)

Copied to clipboard

Shijie Wu, Mark Dredze

Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)

Challenge:	Pretrained contextual representation models have pushed forward the state-of-the-art on many NLP tasks.
Approach:	They propose to use a model that is pretrained on 104 languages for cross-lingual transfer.
Outcome:	The proposed model performs well on 5 NLP tasks covering 39 languages from various language families.

Copied to clipboard

Saurabh Kulshreshtha, Jose Luis Redondo Garcia, Ching-Yun Chang

DeFT-X: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer (2025.findings-emnlp)

Challenge:	Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks.
Approach:	They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting.
Outcome:	The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks.

Copied to clipboard

Sona Elza Simon, Preethi Jyothi

TIPA: Typologically Informed Parameter Aggregation (2026.findings-eacl)

Challenge:	Prior studies have explored multiple approaches to combine task knowledge from task-specific data in a (high-resource) source language with language knowledge from unlabeled text in 'low-resourced' target language.
Approach:	They propose a composable sparse fine-tuning approach that learns task-specific and language-specific sparsen masks to select a subset of the pretrained model's parameters.
Outcome:	The proposed approach performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.

Copied to clipboard

Stef Accou, Wessel Poelman

Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)

Challenge:	Massively multilingual language models enable cross-lingual generalization but underperform on low-resource and unseen languages.
Approach:	They propose a typologically informed framework that constructs proxy language adapters by aggregating existing ones, weighted by typological similarity.
Outcome:	The proposed framework outperforms baselines on five NLP tasks and over 230 languages.

Copied to clipboard

Alan Ansell, Edoardo Ponti, Anna Korhonen, Ivan Vulić

Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021.emnlp-main)

Challenge:	Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners .
Approach:	They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency .
Outcome:	The proposed method outperforms adapters in cross-lingual transfer benchmarks.

Copied to clipboard

Kuan-Hao Huang, Wasi Ahmad, Nanyun Peng, Kai-Wei Chang

BAD-X: Bilingual Adapters Improve Zero-Shot Cross-Lingual Transfer (2022.naacl-main)

Challenge:	Pre-trained multilingual language encoders do not precisely align words and phrases across languages.
Approach:	They propose a learning strategy for training robust models by drawing connections between adversarial examples and failure cases of zero-shot cross-lingual transfer.
Outcome:	The proposed model can achieve good performance even if representations of different languages are not aligned well.

Copied to clipboard

Marinela Parović, Goran Glavaš, Ivan Vulić, Anna Korhonen

On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation (2020.acl-main)

Challenge:	Massively multilingual Transformers (MMTs) have dominated research in multilingual NLP and cross-lingual transfer recently.
Approach:	They propose to learn bilingual language pair adapters (BAs) when the goal is to optimize performance for a particular source-target transfer direction.
Outcome:	The proposed framework improves performance in three standard downstream tasks and for the majority of low-resource languages.

Copied to clipboard

Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, Steffen Eger

ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks (2024.eacl-long)

Challenge:	a standard evaluation setup for supervised machine learning tasks does not hold for natural language generation tasks.
Approach:	They propose to use reference-free machine translation evaluation to compare source texts to system translations to find key limitations.
Outcome:	The proposed metrics perform poorly as semantic encoders for reference-free machine translation evaluation.

Copied to clipboard

Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schuetze

Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU (2022.emnlp-main)

Challenge:	Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding.
Approach:	They propose a prompt-based method for token-level sequence labeling tasks . they propose to decompose an input sentence into single tokens and apply one prompt template to each token.
Outcome:	The proposed method outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer . the method also attains state-of-the-art performance when employed with the mT5 model .

Copied to clipboard

Fenia Christopoulou, Gerasimos Lampouras, Ignacio Iacobacci

Calibrating Zero-shot Cross-lingual (Un-)structured Predictions (2022.emnlp-main)

Challenge:	Current approaches for NLU use CL to improve in-distribution data performance via heuristic-oriented or task-agnostic difficulties.
Approach:	They propose to use CL to improve in-distribution data performance by taking advantage of training dynamics as difficulty metrics instead of heuristic-oriented or task-agnostic difficulties.
Outcome:	The proposed model schedulers improve on in-distribution, out-of-distortion and zero-shot cross-lingual transfer datasets while being 20% faster on average.

Copied to clipboard

Zhengping Jiang, Anqi Liu, Benjamin Van Durme

xGQA: Cross-Lingual Visual Question Answering (2022.findings-acl)

Challenge:	Existing need for model calibration when natural language models are deployed in critical tasks.
Approach:	They compare model calibration methods in a context of zero-shot cross-lingual transfer with pre-trained language models.
Outcome:	The proposed method fails to calibrate more complex confidence estimations in structured predictions compared to expressive alternatives like Gaussian Process Calibration.

Copied to clipboard

Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, Iryna Gurevych

Evaluating morphological typology in zero-shot cross-lingual transfer (2021.acl-long)

Challenge:	a lack of multilingual multimodal datasets has hindered multimodal vision and language modeling efforts.
Approach:	They propose a multilingual evaluation benchmark for the visual question answering task . they extend the established English GQA dataset to 7 typologically diverse languages .
Outcome:	The proposed methods outperform current state-of-the-art models in zero-shot cross-lingual settings, but the accuracy remains low across languages.

Copied to clipboard

Antonio Martínez-García, Toni Badia, Jeremy Barnes

Syntactic Text Processing

Challenge:	morphological typology has been used to improve cross-lingual transfer . however, some language families and typologies consistently perform worse .
Approach:	They examine effects of morphological typology on zero-shot cross-lingual transfer . they perform part-of-speech tagging and sentiment analysis on 19 languages .
Outcome:	The proposed model improves on fusional and introflexive languages, but some language families and typologies perform worse.

Revisiting non-English Text Simplification: A Unified Multilingual Benchmark (2023.acl-long)

Copied to clipboard

Michael Ryan, Tarek Naous, Wei Xu

Paraphrasing

Challenge:	Recent advances in English automatic text simplification have pushed the frontier of multilingual text simulating.
Approach:	They propose to use multilingual evaluation benchmarks to evaluate multilingual text simplification models in English and other languages.
Outcome:	The proposed benchmark outperforms pre-trained models in Russian in zero-shot cross-lingual transfer to low-resource languages.

Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training (2023.acl-long)

Copied to clipboard

Yan Zeng, Wangchunshu Zhou, Ao Luo, Ziming Cheng, Xinsong Zhang

Multimodality

Challenge:	Empirical results show that CCLM significantly outperforms the prior state-of-the-art with an average absolute improvement of over 10%.
Approach:	They introduce a pre-training framework that unifies cross-lingual and cross-modal pre-trained models with shared architectures and objectives.
Outcome:	The proposed framework outperforms the state-of-the-art in two multi-lingual datasets and two multilingual image-text retrieval datasets.

CrossAligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding (2022.findings-acl)

Copied to clipboard

Milan Gritta, Ruoyu Hu, Ignacio Iacobacci

Low-Resource NLP

Challenge:	Task-oriented personal assistants enable people to interact with devices and services using natural language.
Approach:	They propose a method to acquire task knowledge in a high-resource language and then transfer it to the low-resourced language(s) they use unlabelled parallel data to perform a quantitative analysis of the methods.
Outcome:	The proposed methods exceed state-of-the-art (SOTA) scores across nine languages, fifteen test sets and three benchmark multilingual datasets.

Contextual Label Projection for Cross-Lingual Structured Prediction (2024.naacl-long)

Copied to clipboard

Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai-Wei Chang, Nanyun Peng

Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review (2023.acl-long)

Challenge:	Prior work favors simplified label translation or relying on word-level alignments for label projection.
Approach:	They propose a novel approach CLaP which translates text to target language and performs contextual translation on the labels using the translated text as the context.
Outcome:	The proposed approach improves translation accuracy on two prediction tasks and shows 2.4 F1 improvement for EAE and 1.4 F1 for named entity recognition.

Copied to clipboard

Fred Philippy, Siwen Guo, Shohreh Haddadan

CORI: CJKV Benchmark with Romanization Integration - a Step towards Cross-lingual Transfer beyond Textual Scripts (2024.lrec-main)

Challenge:	Pre-trained Multilingual Language Models have shown a strong ability to transfer knowledge across languages.
Approach:	They examine factors contributing to the ability of MLLMs to perform zero-shot cross-lingual transfer . they identify consensuses among studies with consistent findings and resolve conflicts .
Outcome:	The authors outline and discuss factors that contribute to the ability of MLLMs to perform zero-shot cross-lingual transfer.

Copied to clipboard

Hoang Nguyen, Chenwei Zhang, Ye Liu, Natalie Parde, Eugene Rohrbaugh, Philip S. Yu

Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations (2022.emnlp-main)

Challenge:	Naively assuming English as a source language may hinder cross-lingual transfer . despite recent advances in cross-linguistic research, most studies have restricted themselves to two major assumptions .
Approach:	They propose to integrate Romanized transcription beyond textual scripts to capture contact between these languages . they propose to use a benchmark dataset to further encourage in-depth studies of language contact .
Outcome:	The proposed method allows for enhanced cross-lingual representations and effective zero-shot cross-linguistic transfer.

Copied to clipboard

Zhihui Xie, Handong Zhao, Tong Yu, Shuai Li

Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning (2022.coling-1)

Challenge:	Existing studies show that pre-trained ML-LMs can achieve zero-shot cross-lingual transfer without explicit cross-linguistic supervision.
Approach:	They propose a method to remove language-specific factors from multilingual embedding spaces by using a single value decomposition method with multiple monolingual corpora as input.
Outcome:	The proposed method can boost language agnosticism without finetuning . Empirical results show that it consistently leads to improvements over existing models.

Copied to clipboard

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, Dietrich Klakow

Local Byte Fusion for Neural Machine Translation (2023.acl-long)

Challenge:	Multilingual pre-trained language models have shown impressive performance on several downstream tasks for both high-resourced and low-resource languages.
Approach:	They propose to apply multilingual adaptive fine-tuning to 17 most-resourced African languages and three other high-resource languages to encourage cross-lingual transfer learning.
Outcome:	The proposed approach is competitive to LAFT on individual languages while requiring significantly less disk space.

Copied to clipboard

Makesh Narsimhan Sreedhar, Xiangpeng Wan, Yu Cheng, Junjie Hu

Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks (2024.naacl-long)

Challenge:	Existing NLP models rely on a pre-built subword tokenizer to tokenize a sentence . this can be rigid and subwords from low-resource languages are under-represented .
Approach:	They propose a method for byte-based machine translation that aggregates local semantic information.
Outcome:	The proposed method improves on multilingual translation and cross-lingual transfer . it is parameter-efficient and performs competitively to subword models, it is shown .

Copied to clipboard

Nadezhda Chirkova, Vassilina Nikoulina

MAD-G: Multilingual Adapter Generation for Efficient Cross-Lingual Transfer (2021.findings-emnlp)

Challenge:	Existing studies have focused on zero-shot cross-lingual transfer . mBERT, mBART and mT5 provide high-quality representations for texts in various languages .
Approach:	They propose to use mBART and NLLB-200 to finetune a multilingual pretrained language model on input-output pairs in one language and use it to make task predictions for inputs in other languages.
Outcome:	The proposed approach significantly reduces generation in the wrong language with full finetuning and can be competitive in some cases.

Copied to clipboard

Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, Anna Korhonen

ZGUL: Zero-shot Generalization to Unseen Languages using Multi-source Ensembling of Language Adapters (2023.emnlp-main)

Challenge:	Massively multilingual transformers (MMTs) have benefited from additional training of language-specific adapters, but this approach is not viable for the vast majority of languages due to limitations in their corpus size or compute budgets.
Approach:	They propose a multilingual ADapter generation approach which contextually generates language adapters from language representations based on typological features.
Outcome:	The proposed method improves cross-lingual transfer performance on part-of-speech tagging, dependency parsing, and named entity recognition tasks while remaining cost-effective.

Copied to clipboard

Vipul Rathore, Rajdeep Dhingra, Parag Singla, null Mausam

Improving Zero-Shot Cross-lingual Transfer for Multilingual Question Answering over Knowledge Graph (2021.naacl-main)

Challenge:	Existing approaches to zero-shot cross-lingual transfer have focused on training with adapters of a single source and testing either with the target LA or LA of another related language.
Approach:	They propose to leverage LAs of multiple (linguistically or geographically related) source languages for more effective cross-lingual transfer instead of just one source LA . they extend their novel neural architecture, ZGUL, to settings where either (1) some unlabeled data or (2) few-shot training examples are available for the target language .
Outcome:	Extensive experimentation across four language groups, covering 15 unseen target languages, shows improvements of up to 3.2 average F1 points over baselines on POS tagging and NER tasks.

Copied to clipboard

Yucheng Zhou, Xiubo Geng, Tao Shen, Wenqiang Zhang, Daxin Jiang

Frustratingly Simple but Surprisingly Strong: Using Language-Independent Features for Zero-shot Cross-lingual Semantic Parsing (2021.emnlp-main)

Challenge:	Existing approaches to solve multilingual question answering over knowledge graph (KGQA) use of bilingual lexicon induction to map training questions into those in target language circumvents language inconsistency .
Approach:	They propose to use bilingual lexicon induction to map training questions in source and target languages as augmented training data to minimize syntax-disorder.
Outcome:	The proposed model narrows the gap in zero-shot cross-lingual transfer between source and target languages.

Copied to clipboard

Jingfeng Yang, Federico Fancellu, Bonnie Webber, Diyi Yang

Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)

Challenge:	Existing training data is limited for languages other than English, so is the performance of the developed parsers.
Approach:	They propose to apply a pre-trained multilingual model to Italian, German and Dutch parsers where only a small number of manually annotated parses are available.
Outcome:	The proposed model improves on six parsers in English and Italian, German and Dutch, with the addition of universal dependency relations and universal POS tags as model-agnostic features.

Copied to clipboard

Ethan A. Chi, John Hewitt, Christopher D. Manning

Syntactic Text Processing

Challenge:	Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision.
Approach:	They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages.
Outcome:	The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy.

Distilling Efficient Language-Specific Models for Cross-Lingual Transfer (2023.findings-acl)

Copied to clipboard

Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, Ivan Vulić

Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022.acl-long)

Challenge:	Massively multilingual Transformers (MMTs) are widely used for cross-lingual transfer learning.
Approach:	They propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMT for cross-lingual transfer.
Outcome:	The proposed model outperforms models trained from scratch in zero-shot cross-lingual transfer across benchmarks.

Copied to clipboard

Wietse de Vries, Martijn Wieling, Malvina Nissim

Low-Resource NLP

Challenge:	Existing studies on cross-lingual generalisability of large pre-trained models use English training data and test data in unseen languages.
Approach:	They propose to use multilingual pre-trained models to model cross-lingual transfer in a selection of target languages.
Outcome:	The proposed model can be used to improve cross-lingual transfer performance in low-resource languages with no labeled training data.

Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models (2023.findings-emnlp)

Copied to clipboard

Yixin Ji, Jikai Wang, Juntao Li, Hai Ye, Min Zhang

MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer (2021.emnlp-main)

Challenge:	Existing mPLMs can align representations well for myriads of cross-lingual transfer tasks.
Approach:	They propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by anisotropic representations.
Outcome:	The proposed method improves on three zero-shot cross-lingual transfer tasks and over existing methods.

Copied to clipboard

Ilias Chalkidis, Manos Fergadiotis, Ion Androutsopoulos

PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment (2024.emnlp-main)

Challenge:	MULTI-EURLEX is a dataset for topic classification of EU legal documents . fine-tuning a multilingually pretrained model in a single source language leads to catastrophic forgetting of multilingual knowledge and poor zero-shot transfer to other languages.
Approach:	They propose to use the dataset as a testbed for zero-shot cross-lingual transfer to exploit annotated training documents in one language to classify documents in another language.
Outcome:	The proposed model can be used to classify EU legal documents in other languages without a single source language and retain multilingual knowledge.

Copied to clipboard

Jiahuan Li, Shujian Huang, Aarron Ching, Xinyu Dai, Jiajun Chen

XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization (2020.emnlp-main)

Challenge:	Large language models exhibit reasonable multilingual abilities, despite predominantly English-centric pretraining.
Approach:	They propose a framework that establishes multilingual alignment prior to language model pretraining and preserves this alignment using a code-switching strategy during pretraining.
Outcome:	Experiments in a synthetic English to English-Clone setting show that PreAlign outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-linguistic knowledge application.

Copied to clipboard

Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, Mohammad Taher Pilehvar

Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages (2025.coling-main)

Challenge:	Existing evaluation benchmarks for assessing distinct meanings of words are tied to sense inventories, restricting their usage to knowledge-based representation techniques.
Approach:	They propose a multilingual benchmark that models distinct meanings of words in English . they use a binary disambiguation task with gold standards in 12 new languages .
Outcome:	The proposed model can model distinct meanings of words in English even when no tagged instances are available for a target language.

Copied to clipboard

Robert Litschko, Oliver Kraus, Verena Blaschke, Barbara Plank

Probing the Emergence of Cross-lingual Alignment during LLM Training (2024.findings-acl)

Challenge:	lexical gaps between dialects in cross-lingual information retrieval (CLIR) are caused by orthographic variations and different regional expressions.
Approach:	They propose a dataset that consists of seven German dialects extracted from Wikipedia.
Outcome:	The proposed dataset consists of seven German dialects extracted from Wikipedia.

Copied to clipboard

Hetong Wang, Pasquale Minervini, Edoardo Ponti

Don’t Stop Fine-Tuning: On Training Regimes for Few-Shot Cross-Lingual Transfer with Multilingual Language Models (2022.emnlp-main)

Challenge:	Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance.
Approach:	They propose that LLMs can align languages without explicit supervision from parallel sentences without a single linguistic feature.
Outcome:	The proposed model can perform zero-shot cross-lingual transfer even when the vocabularies of two languages have a null intersection, i.e., no tokens are shared.

Copied to clipboard

Fabian David Schmidt, Ivan Vulić, Goran Glavaš

SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition (2022.emnlp-main)

Challenge:	Recent work highlights the fallacies of zero-shot cross-lingual transfer with large multilingual models.
Approach:	They propose to replace sequential fine-tuning with joint fine-uning on source and target language instances.
Outcome:	The proposed techniques yield improved and more stable FS-XLT across the board.

Copied to clipboard

Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer (2024.lrec-main)

Challenge:	Large multilingual models fail to successfully transfer to low-resource languages for zero-shot cross-lingual transfer . sliced fine-tuning for named entity recognition (SLICER) forces stronger token contextualization in the Transformer.
Approach:	They propose a simple yet highly effective approach for improving zero-shot cross-lingual transfer for named entity recognition to low-resource languages.
Outcome:	The proposed approach reduces decontextualization of token representations and classifiers . it yields consistent transfer gains for low-resource languages, the authors show .

Copied to clipboard

Jianyu Zheng, Fengfei Fan, Jianquan Li

Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing (2024.emnlp-main)

Challenge:	Unsupervised cross-lingual transfer is a process of transferring knowledge between languages without explicit supervision.
Approach:	They propose a framework that combines lexical and syntactic knowledge to enhance learning . they use a code-switching technique to implicitly teach lexica and a syntaktic-based graph attention network to help encode syntakic structure.
Outcome:	The proposed framework outperforms baselines of zero-shot cross-lingual transfer with 1.0 3.7 points on text classification, named entity recognition, and semantic parsing tasks.

Copied to clipboard

Deokhyung Kang, Seonjeong Hwang, Yunsu Kim, Gary Lee

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer (2023.findings-emnlp)

Challenge:	Existing approaches to extend semantic parsing (SP) beyond English are challenging due to the complex slot alignment step after translation.
Approach:	They propose a method to enhance cross-lingual transfer for SP by utilizing mPLMs.
Outcome:	The proposed method synthesizes target language utterances from source meaning representations while maintaining high slot value alignment rates.

Copied to clipboard

Fabian David Schmidt, Ivan Vulić, Goran Glavaš

Cross-Lingual Transfer

Challenge:	Existing methods for zero-shot cross-lingual transfer are unreliable due to the lack of pretraining data.
Approach:	They propose to accumulatively average model snapshots from different runs into a single model.
Outcome:	The proposed protocol decouples performance maximization from hyperparameter tuning.

How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning (2023.emnlp-main)

Copied to clipboard

Rochelle Choenni, Dan Garrette, Ekaterina Shutova

Knowledge Enhanced Pre-training for Cross-lingual Dense Retrieval (2024.lrec-main)

Challenge:	Multilingual language models can learn generalisations useful for other languages . yet, it remains unclear to what extent and under which conditions these models benefit from multilingual data and cross-lingual sharing.
Approach:	They propose a training data attribution method to retrieve training samples from multilingual data that are most influential for test predictions in a given language.
Outcome:	The proposed method exploits the ability to learn generalisations useful for other languages on zero-shot cross-lingual transfer for many languages.

Copied to clipboard

Hang Zhang, Yeyun Gong, Dayiheng Liu, Shunyu Zhang, Xingwei He, Jiancheng Lv, Jian Guo

Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)

Challenge:	Existing mPLMs neglect the importance of knowledge in cross-lingual dense retrieval.
Approach:	They propose a novel mPLM that leverages knowledge to learn language-agnostic semantic representations from a multilingual knowledge base and an annotation of Wiki.
Outcome:	The proposed model achieves strong multilingual and cross-lingual retrieval performance with significant improvements over existing mPLMs.

Copied to clipboard

Po-Heng Chen, Yun-Nung Chen